Plato and the Nerd by Lee Edward Ashford;

Author:Lee, Edward Ashford; [ASHFORD LEE, EDWARD]
Language: eng
Format: epub
ISBN: 5065399
Publisher: MIT Press

H(X) is the information we would gain with a perfect observation, and H(X|Y) is the information that is not revealed by the experiment. In other words, H(X|Y) is the remaining randomness after the measurement. The truly astonishing thing about this theorem is that for a wide range of models of measurement noise, the difference H(X) − H(X|Y) represents information that can be encoded with a finite number of bits, even if the original outcome x of experiment X cannot be encoded with a finite number of bits. The information revealed by the experiment, in bits, is finite, although the information in the actual system, in bits, is infinite. In forming the difference H(X) − H(X|Y), both quantities have an infinite offset compared with discrete entropy, but the offsets cancel, and the difference becomes a discrete entropy. This insight is truly remarkable.

For our particular example, where a = 4, we have determined that H(X) = 3. Calculating H(X|Y) precisely is a bit tedious, so I will spare you the details, but our intuition holds up, and H(X|Y) turns out to be slightly less than 0. Hence, C in equation (4) turns out to be slightly larger than 3, indicating that our measurement reveals slightly more than 3 bits of information on average.

It is now worth considering some special cases. Suppose the measurement is perfect. In this case, H(X|Y) is minus infinity because once a measurement is taken, there is no remaining randomness in X. Hence, C is infinite regardless of the value of H(X) (as long as H(X) is not also minus infinity). As a consequence, a perfect observation of a continuous random experiment yields an infinite number of bits of information.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.